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Abstract 


The widespread availability of high-quality Web-based content offers new potential for 
supporting teachers as designers of curricula and classroom activities. When coupled 
with a participatory Web culture and infrastructure, teachers can share their creations 
as well as leverage from the best that their peers have to offer to support a collective 
intelligence or crowdsourcing community, which we dub crowdteaching. We applied a 
collective intelligence framework to characterize crowdteaching in the context of a Web- 
based tool for teachers called the Instructional Architect (IA). The IA enables teachers to 
find, create, and share instructional activities (called IA projects) for their students 
using online learning resources. These IA projects can further be viewed, copied, or 
adapted by other IA users. This study examines the usage activities of two samples of 
teachers, and also analyzes the characteristics of a subset of their IA projects. Analyses 
of teacher activities suggest that they are engaging in crowdteaching processes. 
Teachers, on average, chose to share over half of their IA projects, and copied some 
directly from other IA projects. Thus, these teachers can be seen as both contributors to 
and consumers of crowdteaching processes. In addition, IA users preferred to view IA 
projects rather than to completely copy them. Finally, correlational results based on an 
analysis of the characteristics of IA projects suggest that several easily computed metrics 
(number of views, number of copies, and number of words in IA projects) can act as an 
indirect proxy of instructionally relevant indicators of the content of IA projects. 

Keywords: Distributed learning environments; evaluation of CAL systems; interactive 
learning environments; pedagogical issues 
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Introduction 


Teachers have long been designing and modifying curricula and lesson plans (Ball & 
Cohen, 1996; Brown & Edelson, 2003; Fogleman, McNeil, & Krajcik, 2011; Remillard, 
2005). More recently, this phenomenon, called teachers as designers, has drawn 
renewed interest (e.g., Davis & Varma, 2008), prompted in part by the widespread 
availability of high-quality online resources, called open educational resources (OER), 
via the Internet. As OER become increasingly and widely available, research is needed 
to understand how teachers design curricula, lesson plans, and classroom activities 
using such resources. In particular, in networked computing environments that provide 
easy sharing and reuse, how do teachers become engaged in designing, sharing, and 
modifying instructional artifacts, and do these activities impact their resulting quality? 

Our approach for supporting teachers as designers is via a free, Web-based authoring 
tool called the Instructional Architect (IA.usu.edu), which enables teachers to find and 
design instructional activities for their students using OER (Recker, 2006). Teachers 
can share these resulting activities, called IA projects, by making them publically 
available within the IA. These IA projects can then be viewed, copied, or adapted by 
other IA users to further support their own teaching activities (Recker et ah, 2005; 
Recker et ah, 2007). Viewed in this way, the LA provides an infrastructure for collective 
intelligence and crowdsourcing, which we dub crowdteaching, in which teachers can 
create, share, and iteratively adapt instructional activities using OER, leveraging from 
their peers’ work to best serve the needs of their students (Benkler, 2006; Borgman et 
ah, 2008; Porcello & Hsi, 2013). 

In collective intelligence communities, loosely organized groups of people connected by 
the Internet work together to accomplish tasks in ways that appear more intelligent, 
more effective, and more efficient than working alone (Malone, Laubacher, & 
Dellarocas, 2009). However, studies of collective intelligence sites, such as Wikipedia, 
suggest that these peer production models may succeed only when they are aimed at 
focused tasks and coupled with incentives to harness the work of the best contributors 
(Malone et ah, 2009). Thus for crowdteaching models to succeed, we need more 
nuanced understandings of how teachers can participate in such environments to create 
and share instructional activities around OER. 

The purpose of this exploratory article is to explore how teachers’ activities within the IA 
may reveal crowdteaching processes. We first provide an overview of the IA system, and 
then describe how it fits within a framework for examining collective intelligence 
communities. Next, we examine the usage and creation activities of two teacher samples 
and analyze the characteristics of a subset of IA projects implemented by teachers in 
classrooms. In this way, we explore what aspects of these teachers’ design activities may 
enhance the collective intelligence of the IA community and the nature of the artifacts 
that are produced. 
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Collective Intelligence Context: The Instructional Architect 


The collective intelligence context for this work is the Instructional Architect 
(IA.usu.edu), in use since 2001. The IA is a free, easy-to-use Web-based authoring tool 
that enables teachers to easily find and assemble OER into learning activities for their 
students (Recker, 2006). The IA was originally designed for K-12 teachers, and has been 
widely used in various subject areas (e.g., art, engineering, math) by teachers in both K- 
12 and higher education settings. 

To support teachers’ design and collaboration activities, the IA offers several functions. 
For example, the “My Resources” area allows teachers to search for and save OER to 
their personal collections. The “My Projects” area allows teachers to create IA projects 
using collected OER, and publish (or share) these IA projects. From the perspective of 
collaboration, teachers can view published IA projects by using the “search” function, 
and copy their favorite ones to their personal collection. In this way, the IA supports 
crowdteaching processes in that teachers can create and contribute LA projects, as well 
as view, copy, and build upon other teachers’ IA projects that have been contributed to 
the IA community. 

Since 2005, the IA has attracted over 7,600 registered users, who have gathered 
approximately 74,000 OER and created over 17,300 IA projects. Since August 2006, 
public IA projects have been viewed over 2.5 million times. These IA projects address a 
range of subject areas and grade levels, rely on a range of pedagogies, and incorporate 
varying numbers of OER in many different ways. 

Figure 1 shows an example IA project created by a middle school science teacher. In this 
example, the teacher has written text that presents a problem, as well as supporting 
information to help solve the problem. This information includes a link to an online 
simulation that helps students learn what variables affect density. Here, the OER is 
included as a support to the primary problem solving activity. 

Figure 2 shows a simpler IA project on a science-related topic for 8th grade students. In 
this example, the teacher directs students to a link, while asking them to complete a 
worksheet. In this case, the OER plays the major role in instruction. 
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Problem Presentation: 

You and your Mends have been camping As you leave the camp site, you discover a vory large tree has fallon on the trail. It is too largo to climb over or go around. So. you 
take a different route, which leads you to a large lake. The only way to get to the other side is to go across it. Since there are no rafts around, you realize will need to build one. 
You see around you: bricks, logs. long, thick trco branches, metal tent poles, and shovels. Last year you learned about density and what helps objects float You now need to 
figure out what objects will float and how density works. 



Current Knowledge: 

What are some things you already know about density in order to solver this problem? How oould this Information help you solve the problom about building a raft? 

Information Sources: 

You will need to gather more information about density so you can solve the problem of building the raft. 

Click on the link to discover more about density. Interactive Donsity Gamo 

Continue with the next website that will give you more information about what density it and how to calculate density. Calculating Density Interactive Page 

Information Seeking: 

You should now have enough Information to sohre the problom with your Mends at the lako and bo ablo to build a raft Howover. If you would like to read more about donsity 
on your own, you may do your own internet search. You can either use Google, or open another tab use a search engine of your choice. Google website 

What Have You Learned? 

At the start of this website, you wore asked what you already know about donsity. 1 would like you to now shore what you havo learned about donsity by doing this activity 

What did you not know about density before that you know now? 

What resource helped you best understand what density is? 

Do you feel you have a better understanding of what density Is? 

Solution: 

Your job now is to describe the best solution to the problem about building a raft. What materials are best to use and why? Are there objects that should NOT be used? 
Carefully explain your solution and why that is the best solution for this problem. 

Reflection: 

How did this activity better help you understand density? 

Were you able to come up with a reasonable solution to the problem? 

Explain how this activity will help you remember donsity in the future. 


Figure l. Example of an inquiry-based IA project. 



Forces of Motion 


Answer the questions on a sheet of paper 


http://teachertech.rice.edu/Participants/louviere/Newton/ 


Figure 2. Example of an IA project with minimal text and a link to one OER. 


|Vol 15 | N04 


Creative Commons Attribution 4.0 International License 


Sept./14 


141 















Crowdteaching: Supporting Teaching as Designing in Collective Intelligence Communities 


Recker, Yuan, and Ye 


Theoretical Framework 


In this section, we first review a framework for characterizing collective intelligence 
communities and show how it applies to crowdteaching in the context of the 
Instructional Architect. We then describe two indicators for examining and 
characterizing the content of teachers’ IA projects. 

Collective Intelligence 

In a recent paper, Malone et al. (2009) analyzed over 250 examples of online collective 
intelligence and crowdsourcing communities, using two sets of related questions. The 
first set of questions examines “what,” that is, the goals or outcomes of the collective 
intelligence community. For example, key activities may involve creating artifacts or 
deciding on winners. These questions also address the primary processes behind these 
activities, for example, collecting or collaborating. 

The second set of questions addresses “who” is engaged in tasks. In a collective 
intelligence community, either an egalitarian crowd or a hierarchy (where some 
participants have more decision-making power than others, e.g., editors in Wikipedia) 
performs the tasks. These questions also address “why,” or user motivations and 
incentives for engaging in tasks. In some cases, money may be the motivator. For 
example, in TeachersPayTeachers.com, teachers post lesson plans and activities that 
then can be purchased and revised by others (Abramovich & Schunn, 2012). In other 
communities, however, altruism is a key driver. In the Tapped In online community, for 
example, educational professionals engaged in mentoring and discussions with no 
concrete reward structures (Farooq, Schank, Harris, Fusco, & Schlager, 2007). 

Table 1 

Create and Decide Dimensions from Malone et al. (2009) Applied to the Instructional 
Architect 



What 

Who 

Why 

How 

Create 

IA project 

Teachers, 

individually 

Motivate students; 
supplant and supplement 
textbook; increase 
efficiency 

Create personal 
collection of IA 
projects 

Decide 

View IA 
project 

Teachers, 

individually 

Leverage wisdom of 
crowd, learn from peers 
and OER 

View public IA 
projects 

Decide 

CopyIA 
project 

Teachers, 

individually 

Leverage wisdom of 
crowd, improve efficiency 
and effectiveness 

Copy public LA 
projects to personal 
collection 
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Following the framework of Malone et al. (2009), Table 1 shows key create and decide 
dimensions in the IA in terms of “what is being accomplished (goal),” “who is 
performing the task,” “why they are doing it (incentives),” and “how it is done.” For the 
create dimension in the IA, teachers work independently to design IA projects. As 
reported elsewhere, we have found that key motivations for teachers in using the IA 
include the desire to increase student motivation by using interactive content, to 
supplement their textbook materials, to increase student understanding using 
interactive resources, and to increase their efficiency as teachers (Recker et al., 2005; 
Recker et al., 2007). Teachers accomplish these tasks by creating a collection of IA 
projects, which they can then choose to share with only their students or with anyone 
using the IA site. 

In the decide dimension in the IA, teachers can individually decide to view or copy an 
existing IA project from the public collection. For example, a teacher might decide to 
search for and view IA projects on a particular topic to see how other teachers are 
choosing to teach it and what OER they are using to support student learning. This 
approach might be more efficient than an unconstrained search of the Internet, which 
many teachers find highly inefficient (Mardis, El Basri, Norton, & Newsum, 2012). It 
might also be more effective in that teachers can learn from other teachers and from the 
content of the OER they select (Ball & Cohen, 1996; Davis & Krajcik, 2005; Drake & 
Sherin, 2006). If teachers especially like an IA project, they can decide to copy it to their 
personal collection for further editing, adaptation, and reuse. 

Examining and Characterizing Artifact Content 

A key objective of the collective intelligence process is to leverage the work of others to 
improve the effectiveness and/or efficiency of creating artifacts. In education, the rapid 
rise of repositories of OER, such as those provided by TeacherTube, the National 
Science Digital Library (NSDL.org), the Khan Academy, the OER Commons, and 
OpenCourseWare (Atkins, Brown, & Hammond, 2007), has made evaluating the 
content of OER more pressing (Sumner, Khoo, Recker, & Marlino, 2003; Kastens et al., 
2005; Porcello & Hsi, 2013). 

Characterizing the content of OER, however, has proven a complex task. Multiple 
factors can impact a person’s judgment, such as an OER’s availability, credibility, 
currency, and authority of the content, and the context (e.g., pedagogy, setting) of use of 
an OER (Custard & Sumner, 2005; Leary, Giersch, Walker, & Recker, 2009; Wetzler et 
al., 2013). In this work, we adopted strategies used in previous work that assessed 
teachers’ naturally occurring artifacts (e.g., lesson plans) as a means to measure the 
quality of students’ learning opportunities (Penuel & Gallagher, 2009). Thus, we 
examined the content of artifacts created by teachers (i.e., IA projects) using two sets of 
instructionally relevant indicators. These indicators are described next. 

Indicator 1: Problem-based learning (PBL). We applied a rubric developed in previous 
research (Walker et al., 2012) to score an IA project’s alignment with a form of inquiry 
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learning, specifically problem-based learning (Barrows, 1996). The rubric consisted of 11 
elements in four categories (Authentic Problem, Learning Processes, Facilitator, and 
Group Work), with the presence of each element rated on a 0-2 scale, resulting in a 
maximum possible score of 22 points. Three raters, randomly selected from a pool of 
five raters, independently scored the PBL alignment of the LA projects. Overall inter¬ 
rater reliability of the rubric as measured by average one-way random effects intra-class 
correlation (ICC) was high, with ICC = .86 (Walker et al., 2012). 

Indicator 2: Offload-adaptation-improvisation (OAI). As an outcome of studying 
teachers’ adaptation of an innovative curriculum, Brown and Edelson (2003) devised 
the design capacity for enactment (DCE) framework within a “teaching as design” 
paradigm. As part of the framework, they defined a continuum of curriculum use, 
ranging from offloads to adaptations to improvisations. This continuum describes the 
distribution of responsibility for instruction between the teacher and the curriculum. In 
particular, in offloads, the curriculum is implemented essentially unchanged and the 
bulk of instructional decisions are contained in the instructional materials. In 
improvisation, the teacher flexibly borrows and customizes pieces while playing a major 
role in the decision-making process. The adaptation category represents the midpoint of 
the continuum. 

Building on our previous work (Recker et al., 2007), we operationalized aspects of the 
Brown and Edelson continuum to characterize how LA projects integrate OER (see Table 
2). To measure inter-rater reliability, one coder scored all 72 IA projects and a second 
coder scored a random subset of IA projects. Krippendorff s alpha suggests moderate to 
high reliability (Kalpha = .69). 

Table 2 


Rubric for Measuring OER Integration (OAI Scale) (Recker et al., 2007) 


Score 

Name 

Definition 

1 

Offload 

Teachers provide links to OER with little additional teacher- 
created instructional guidance (e.g., no explanations or 
instructions). Use tends toward lists of links (perhaps with 
added navigational information). For example, see Figure 2. 

2 

Adaptation 

A midpoint, with only some of the elements listed below. 

3 

Improvisation 

Teachers link to OER as a starting point or reference but have 
clearly designed their own instructional elements, for example 
by including learning goals, instructional activities, 
descriptions of OER use, or assessment items. For an example, 
see Figure 1. 
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Research Design and Methods 


The core research questions guiding this work follow the framework of Malone et al. 
(2009) and are thus organized around users (teachers) and artifacts (IA projects): 

1. How do teachers engage in collective intelligence activities within the IA? 

2. How do teachers’ collective intelligence activities and their collective 
intelligence artifacts relate to a) the creation of inquiry-based IA projects, 
and b) the integration of OER within IA projects? 

Participants and Sampling Procedures 

To investigate these two questions, we analyzed two different datasets corresponding to 
each of the questions. These datasets illustrate LA usage in two different settings. The 
first sample, “in the wild”, includes users who were not actively recruited to use the IA. 
The second sample, PD participants, was drawn from a teacher professional 
development (PD) opportunity centered around use of the IA conducted in a western 
U.S. state. 

The sampling process for “in the wild” teachers was we selected users who created an IA 
account during the 2009 calendar year and also indicated their years of teaching 
experience in the optional portion of their profile when they registered with the IA. 
These 200 teachers created a total of 520 IA projects. 

The second sample, PD participants, was comprised of 36 middle school mathematics 
and science teachers who participated in a professional development series lasting three 
months. The professional development (PD), described elsewhere (Walker et al., 2012), 
focused on enhancing teachers’ technology skills for finding and selecting OER, and 
designing classroom activities around these OER using the Instructional Architect. 
These teachers created a total of 351 IA projects from September 2010 to August 2011. 
Using teacher journals, we selected two projects from each teacher (72 total) for further 
analysis using the following criteria: 1) the IA projects were created during the PD 
training period, 2) they were implemented in their classroom teaching, and 3) they were 
scored using the PBL alignment and OAI rubrics. As these teachers were all part of a 
sustained PD experience and those 72 projects were implemented in teaching, teacher 
behaviors and their IA project features are likely different from those of teachers “in the 
wild.” 

Data Sources 

For these two samples of users, several sources of data were analyzed. First, users 
completed an optional user profile upon creating their free IA account, in which they 
were asked to rate their comfort level with technology on a Likert scale, and report their 
years of teaching experience. Second, the IA was instrumented to automatically collect 
and aggregate detailed usage data, as well as, third, IA project features (Khoo et al., 
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2008). In addition, two IA projects created by each of the 36 PD participants (72 total 
IA projects) were hand scored in terms of their PBL alignment score and their OAI score 
(described above). 

Data Analysis 

To address RQi, we used descriptive statistics to explore IA usage from both teacher and 
IA project perspectives. Table 3 shows definitions for the variables examined in the 
study. 

To address RQ2, correlation analyses were conducted to examine the bivariate 
relationship between the IA project variables (PBL or OAI score) and IA usage variables. 
Since the PBL alignment scores were not normally distributed and the OAI score was a 
categorical variable, Spearman ranked correlations were used. To further investigate 
what variable(s) predict the OAI scores, multinomial logistic regression models were fit. 
Before running the regression model, multicollinearity was examined according to the 
bivariate correlation analysis results, and no high correlations (> .8) were identified. A 
series of models and predictor variables were tested using the backward elimination. 
The final models were selected based on their overall significance and parsimony. 

Table 3 

Definition oflA Usage Variables 



IA usage variable 

Definition 


# of logins 

# of times a teacher logs into the IA system 


# of OER used in all IA 

# of Open Educational Resources used in all IA 


projects 

projects 


# of IA projects created 

# of IA projects created 

-L LdLIlLI 

activities 

# of public IA projects 
created 

# of public IA projects created 


# of IA projects copied 

# of IA projects an IA user completely copied 


from others 

from other IA projects 


# of IA projects viewed 

# of IA projects an IA user viewed except the ones 
created by himself or herself 


# of words 

# of words in the IA project content area, 
excluding the words in the overview area and 
the links 

IA 

# of links 

# of links used in an IA project 

projects 

# of edits 

# of times the IA project is edited by its author 

features 

# of times viewed * 

# of times a project has been viewed by all other 
users 


# of times copied* 

# of times a project has been copied by all other 
users 


Note. “Only teacher-averaged IA project features are available in the dataset for “in the 
wild” users, therefore “# of times viewed” or “# of times copied” are not reported. 
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Results 


Research Question 1: Teacher Collective Intelligence Activities 

The first research question explored organic teacher activity and how it might relate to 
teachers’ collective intelligence processes. Table 4 shows summary data for the 200 “in 
the wild” teachers who created an IA account over the course of one year, and the 520 IA 
projects they created. On average, these teachers created a small number of IA projects 
and chose to share almost two thirds of them. In addition, 15% of their IA projects were 
copied directly from other IA projects. Thus, these teachers can be seen as both 
contributors to and consumers of crowdteaching. 

Further, the distribution of most usage values is skewed and follows a Zipf (or long tail) 
distribution (Anderson, 2006; Recker & Pitkow, 1996). As is common in many Internet- 
based datasets, a small number of users account for a majority of the activity (see Figure 
3 for an example of a histogram of teachers’ number of logins). This has also been called 
the “90-9-1” rule or “participation inequality,” in that in typical online communities, 
90% of participants are lurkers, 9% contribute occasionally, and 1% account for the 
most contributions (Nielsen, 2006). 

Table 4 

Descriptives of Users’ (N = 200) Activity and Their IA Project Features (Data Collected 
over a l-Year Period) 



Variable 

Mean 

Median 

SD 

Min 

Max 


# of logins 

10.38 

7 

10.59 

1 

57 


# of OER used in all IA 
projects 

16.82 

10 

24.02 

0 

217 


# of IA projects created 

2.60 

2 

2.04 

1 

10 

Teacher 

activities 

# of public IA projects 
created 

1-73 

1 

1-95 

0 

10 

(N = 200) 

# of IA projects copied 
from others 

•58 

0 

1.46 

0 

9 


% of IA projects copied 
from others 

15.22 

0 

29.50 

0 

100 


# of IA projects viewed 

12.98 

7 

17.44 

0 

134 

IA projects 

# of words 

186.02 

94 

308.02 

0 

2692 

features (N = 

# of links 

4-23 

3 

4.17 

5 

28 

520) 

# of edits 

2.87 

2 

3-29 

0 

21 


Note, ‘o OER used in all IA projects’ denotes teachers who did not use any OER in their 
IA projects; ‘o words’ denotes IA projects with only link(s) and without any words in the 
project content area; ‘o links’ denotes IA projects with no links; ‘o edits’ denotes IA 
projects without modification after creation. 
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Figure 3. Histogram of teachers’ number of logins over a l-year period. 



Figure 4. Teachers’ creation activity categorized by their comfort level with technology. 
(For PD participants, comfort level with technology ranges from “o” to “4” levels; “o” 
refers to “novice” and “4” refers to “expert”. However, due to a change in the interface 
“in the wild” users reported their comfort levels on “1” to “3” scale.) 
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Teaching Experience (years) 


♦ AVG. # of IA projects 
created 


■ AVG. # of public IA 
projects created 


* AVG. # of IA projects 
copied from others 


Figure 5. Teachers’ creation activity categorized by their teaching experience. 


Figures 4 and 5 show these teachers’ IA project creation activity categorized by their 
self-reported comfort levels with technology and their years of teaching experience. 
Figure 4 shows that teachers who had a higher comfort level with technology appeared 
to have more contributor behavior: They created and shared the most IA projects but 
copied less. Teachers who had a lower comfort level with technology tended to display 
more consumer behavior: They created and shared less IA projects but copied more. 
Figure 5 shows teachers’ usage activity categorized by their self-reported teaching 
experience. Here, no clear pattern was apparent. 

In sum, teachers can be involved in crowdteaching in different ways: offering wisdom by 
making their IA projects public, and benefiting from others’ wisdom by viewing and 
copying IA projects. In terms of enhancing the collective intelligence of the community, 
the challenge then becomes automatically deriving metrics that help identify IA projects 
that are valued in the community. This challenge is addressed next. 

Research Question 2a: Problem-Based Learning Alignment 

This research question examines the relationship between IA project features, teacher 
characteristics, and the creation of inquiry-oriented IA projects. Table 5 shows summary 
data for the 36 teachers and their 72 LA projects for data collected over one year. 

In comparison to those IA projects created by teachers “in the wild,” these IA projects 
showed a much higher average number of edits, suggesting higher levels of effort. In 
addition, perhaps because these teachers were part of a sustained PD intervention, these 
teachers also showed, on average, higher levels of IA activity, including number of 
logins, IA project created, projects viewed, projects copied, and OER collected. 
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Table 5 

Descriptives of PD Participants’ (N = 36) Activity and IA Project Features (N = 72) 
(Data Collected over a l-Year Period) 



Variable 

Mean 

Median 

SD 

Min 

Max 

Teacher 

activities 

(N= 3 6) 

# of logins 

# of OER used in all IA projects 

# of IA projects created* 

# of public IA projects created 

# of IA projects copied from 
others 

% of IA projects copied from 
others 

# of IA projects viewed 

31.42 

33-72 

9-50 

4.08 

2.25 

25-49 

20.39 

27 

23 

7 

3 

1 

18.33 

21 

28.02 

31-39 

9.07 

7.98 

2.78 

25.78 

11.86 

6 

8 

2 

0 

0 

0 

2 

179 

179 

57 

49 

12 

80 

52 


PBL Alignment Score 

3-32 

2 

3-45 

1 

17 


# of words 

169.86 

113-50 

168.28 

9 

859 

IA project 

# of links 

5-36 

4 

4-50 

0 

37 

(N=72) 

# of edits 

69.94 

51 

6370 

5 

388 


# of times viewed (N = 51)** 

336.84 

199 

391-96 

13 

1995 


# of times copied (N = 51)** 

0.47 

0 

•95 

0 

4 


Note. * A total of 351 IA projects were created and 72 were selected for detailed analysis. 
** Only public IA projects can be viewed and copied. 


Of these 72 IA projects, 51 (70.83%) were made public by their authors, meaning they 
could be viewed and copied by others. Table 5 shows that the 51 public IA projects were 
viewed more frequently than they were copied, suggesting that the IA users preferred to 
view IA projects for ideas and OER rather than to completely copy them. Copying an IA 
project may be an indication that a teacher places a very high value upon it. Table 5 also 
shows that overall mean scores for PBL alignment were low and their distribution was 
skewed (M = 3.32 on a 22-point scale). This skew precluded the use of statistics 
assuming normality, such as Pearson’s correlation and multiple regression. 

Spearman’s rank order correlation coefficients were calculated to investigate the 
relationships between various IA project features and PBL alignment scores (see Table 
6). Results suggest that PBL alignment scores are positively and moderately correlated 
with the number of words and the number of views. Thus, if the PBL alignment score is 
viewed as an indicator of a useful IA project, the number of views and words can act as 
an indirect proxy of this measure. Table 6 also shows positive and moderate correlations 
between the number of words and the number of links, edits, and views. Moderate 
positive correlations were also found between the number of times an IA project was 
edited and the number of links and views. 

Spearman’s rank correlations were used to explore the relationships between teacher 
characteristics and the PBL alignment score of their IA projects. Table 7 shows that 
teachers’ reported teaching experience was moderately and negatively correlated with a 
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variety of IA activities, such as the number of logins, the number of OER used, the 
number of IA projects created, the number of public IA projects created, and the 
number of the IA projects viewed, suggesting that novice teachers may have found 
certain features of the IA more useful. Additionally, Table 7 shows a positive correlation 
between teachers’ comfort level with technology and the number of public LA projects 
created. Thus, as teachers’ comfort level with technology increases, they tend to publish 
and share more IA projects. This is similar to the pattern identified in Figure 4. 

Other moderate and positive correlation results are not surprising and suggest that an 
engaged user shows overall higher levels of activity on all IA features. Note, however, 
that no strong correlations were found between the PBL alignment score and teacher 
characteristics. 

Table 6 

Spearman’s Rank Correlations between PBL Alignment Score, OAI Score, and IA 
Project Features for IA Projects Created by PD Participants 


Variable 

1 

2 

3 

4 

5 6 

1. PBL alignment score 






2. IA project OAI score 

.30* 





3. #of words 

.42** 

.60** 




4. # of links 

.08 

•23 

.27* 



5. # of edits 

•19 

■43** 

•52** 

.38** 


6. # of times viewed (IV= 51) 

.38** 

•25 

.28* 

.26 

•37** 

7. # of times copied (IV=5i) 

.09 

•36* 

.001 

-•05 

-.08 .19 


* p < .05; ** p < .01. 
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Table 7 

Spearman’s Rank Correlations between PBL Alignment Score, OAI Score, and 
Characteristics of PD Participants 


Variable 

1 

2 

3 

4 

5 

6 

00 

cO 

1. PBL alignment 








score 








2. IA project OAI 

.30* 







score 








3. Teaching 
experience 

-.22 

-.14 






4. Comfort level 
with technology 

.10 

-.07 

-.08 





5. # of logins 

-•05 

.24* 

_ g c~y 

-•23 




6. # of OER used 
in all IA projects 

•15 

.27* 

-.41** 

-.14 

.30** 



7. # of IA 
projects created 

-.06 

-.003 

-•33** 

-.02 

•58** 

.60** 


8. # of public IA 
projects 

-.08 

.07 

-•35* 

.30** 

-.02 

.30** 

.18 

9. # of times 
user copied IA 
project 

.20 

.18 

-.07 

.07 

-•03 

.51** 

•37** -09 

10 .# ofIA 
projects viewed 

-.09 

-.07 

-•34** 

•13 

.40** 

•39** 

.52** .15 .17 


*p < .05; ** p < .01. 


Research Question 2b: Use of OER in IA Projects 

This question examines the relationship between IA project features, teacher 
characteristics, and how IA projects are designed to integrate OER. For these 72 IA 
projects, 23 (31.9%) of the IA projects were categorized as an offload, 38 (52.8%) IA 
projects were in the adaptation category, and 11 (15.2%) IA projects were in the 
improvisation category (see Table 2 for definitions). 

Spearman's rank order correlation coefficients were calculated between OAI scores and 
IA project features (see Table 6). Results show that OAI score is positively correlated 
with the number of words, the number of edits, and the number of times an IA project 
was copied. Thus, as the number of words, edits, and copies of an IA project increased, 
its OAI score increased toward the improvisation end of the scale. 

Spearman's rank order correlation was also used to investigate the relationships 
between OAI scores and teachers’ characteristics (see Table 7). Results suggest that OAI 
scores are significantly and positively correlated with teachers’ number of logins and 
number of OER used. Thus, as a teacher’s number of logins and number of OER used 
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increased, the OAI score of their IA projects increased toward the improvisation end of 
the scale. 

As noted, Tables 6 and 7 show that the OAI scores of the 72 IA projects are significantly 
and positively correlated with the number of words, the number of edits, the number of 
times an IA project was copied, the number of logins, and the number of OER used. 
Several of multinomial logistic regression models were fit to test whether different 
combinations of these five variables could significantly predict the OAI scores of IA 
projects. Note that multicollinearity was examined and eliminated as a concern for this 
dataset. Two final models are reported as follows. 

The first multinomial logistic regression model (see Table 8) was based on the 72 IA 
projects with four predictor candidates. The results show that only the number of words 
is a significant predictor of OAI score. This means that after holding other variables 
constant, for each unit increase in the number of words, the multinomial odds ratio for 
an IA project in the adaptation category (OAI = 2) relative to the offload category (OAI = 
1) would be expected to increase by 2%. Similarly, after holding other variables constant, 
for each unit increase in the number of words, the multinomial odds ratio for an IA 
project in the improvisation category (OAI = 3) relative to the offload category (OAI = 1) 
would also be expected to increase by 2%. 

Table 8 


Multinomial Logistic Regression on OAI Score for 72 IA Projects 


Variable 

Adaptations (OAI = 2) 

Improvisation (OAI = 3) 

OR 

Wald x 2 P 

OR 

Wald x 2 

P 

# of words 

1.02 

7.21 .007 

** 1.02 

8-54 

. 003** 

# of edits 

1.01 

1.56 .21 

1.02 

3.10 

.08 

# of logins 

1.02 

•49 -49 

1.02 

•37 

•54 

# of OER used in 
projects 

all IA 1.01 

.09 .76 

1.00 

•003 

.96 

Note. The reference category is offloads (OAI = i).**p < 

.01. 



Table 9 






Multinomial Logistic Regression 

on OAI Score for 51 Public IA Projects 


Variable 

Adaptations (OAI = 2) 

Improvisation (OAI 

= 3) 

OR 

Wald x 2 p 

OR 

Wald x 2 

P 

# of times copied 

1.07 

.02 .89 

2.69 

4.00 

.046* 


Note. The reference category is offloads (OAI = 1). * p < .05 
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The second multinomial logistic regression (see Table 9) was based on the 51 public IA 
projects and one predictor was retained in the final model. The results indicate that the 
number of times users copied IA projects is a significant predictor of OAI score. For 
each unit increase in the number of times users copied an IA project, the multinomial 
odds ratio for a project in the improvisation category (OAI = 3) relative to the offload 
category (OAI = 1) would be expected to increase by 169%. However, the comparison 
between the adaptation category (OAI = 2) and the offload category (OAI = 1) did not 
show this trend. 

In sum, among the five variables that are significantly correlated with the OAI scores of 
the IA projects, only the number of words and the number of times an IA project was 
copied were significant predictors of OAI score. 


Discussion 


The evidence suggests that collective intelligence (called crowdteaching) activities occur 
within the IA. Teachers, on average, chose to share almost two thirds of their created 
artifacts, while a small proportion of their IA projects were copied directly from other IA 
projects. Thus, these teachers can be seen as both contributors to and consumers of 
crowdteaching processes. In addition, IA users preferred to view IA projects rather than 
to completely copy them. This suggests that they may be browsing for ideas or finding 
only a smaller set of IA projects that completely meet their needs. Copying an IA project 
suggests a higher level of endorsement of its content. Finally, PD participants showed 
overall greater levels of LA activity, compared to those teachers “in the wild.” Thus, like 
findings from other research of loose online communities (Abramovich & Schunn, 2012; 
Nielsen, 2006), participation shows deep inequalities, but can be nurtured. 

In examining the possible influences of teacher characteristics, we noted similar 
patterns in the two samples of teachers in terms of their reported comfort level with 
technology. “In the wild” teachers with lower levels of comfort in technology use 
appeared to display more consumer behaviors, while higher level teachers appeared to 
show more contributor behaviors. A similar pattern is also evident in the PD 
participants: Teachers’ comfort level with technology was positively (and moderately) 
correlated with sharing behaviors. In terms of the teaching experience, patterns were 
different. “In the wild” teachers showed no clear pattern; for PD participants, teaching 
experience was negatively (and moderately) correlated with IA project creation and 
sharing. We also acknowledge that as self-reported data, these may not capture the most 
important underlying constructs. 

In addition, a goal was to find proxy variables that could be easily computed and that 
also aligned to instructionally relevant indicators. Distilling such variables could 
support IA users (teachers) in quickly identifying useful IA projects to either use or 
further adapt. We coded IA projects implemented in classrooms by teachers using two 
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indicators, the problem-based learning rubric and the OAI scale, and examined their 
relationship with several usage metrics. Correlational results suggest that among several 
metrics related to two instructional indicators, the number of words in IA projects was 
the best indirect proxy of both indicators. 

It is possible that IA projects with more content (i.e., words) would necessarily score 
higher on these two indicators. However, it is unclear that simply having more textual 
content would cause raters to find greater evidence of inquiry learning elements, or 
improvisation around OER. Similarly, a comprehensive study of Wikipedia article 
quality also found that the number of words was the most robust (and easiest to extract) 
predictor of quality although there is no reason to expect that lengthy encyclopedia 
articles are necessarily better (Blumenstock, 2008). 

We note that this study had several limitations. The first is the small number of users 
and IA projects analyzed against the larger backdrop of users and usage. True 
crowdsourcing environments typically assume an Internet-scale community behind 
crowdsourcing activities (Estelles-Arolas & Gonzalez-Ladron-de-Guevara, 2012). In our 
case, we restricted our analyses in the first sample to users who clearly identified 
themselves as teachers, which severely restricted the user base. Because we relied on 
these users voluntarily completing a brief demographic survey, we may have 
unintentionally excluded actual teachers. In the second sample, however, we 
deliberately only included IA projects that teachers told us were used in actual 
classroom practice. In the future, instead of using a restricted sample, a more systematic 
sampling process should be employed and more comprehensive teacher demographic 
information should be collected. Finally, we relied upon two indicators of instructional 
quality. It is certainly possible that others could be identified that could serve as more 
robust indicators. 


Conclusion 


This article explored collective intelligence processes in the context of a Web-based tool 
for teachers, the Instructional Architect, to support teachers as designers using OER. 
Our analyses were guided by a framework, which posits a very loose notion of 
community wherein members may have very diffuse interactions with one another 
(Malone et ah, 2009). The IA community is in that vein. 

Results of the study have implications for both research and practice in the OER 
community. From the research perspective, this study reveals different patterns in 
terms of how teachers engage with OER: They can be contributors by creating and 
sharing new content; or they can be consumers by viewing and even copying other 
teachers’ content. In addition, the study identified various factors that influenced 
teachers’ engagement with the IA, such as their teaching experiences, comfort levels 
with technology, and whether or not they attended professional development 


|Vol 15 | N04 


Creative Commons Attribution 4.0 International License 


Sept./14 


155 






Crowdteaching: Supporting Teaching as Designing in Collective Intelligence Communities 


Recker, Yuan, and Ye 


workshops. Moreover, the analysis of user activities indicates that the frequency of 
teacher activities in the IA often followed the “long tail” distribution, which provides 
further evidence for Nielsen’s (2006) participation inequality rule. 

In terms of practice, results from the study have implications for system design. For 
example, the “decide-how” dimension is largely done individually, and supports for this 
step within the IA are mostly implicit. As such, scaffolds in the IA interface could be 
designed to better represent user activity (e.g., sorting search results by identified 
proxies of quality, including number of words, views, or copies) in order to help teachers 
better leverage crowd wisdom. 

Future work includes developing means for automatically conducting longerterm 
analyses of the activities of IA users, as well as the evolution of their artifacts. We are 
also developing computational approaches that can scale to study how the micro-level 
activities of these users and their designs might affect the macro-level behaviors of the 
community as a whole (Walker et al., 2011). 
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